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Abstract 

The commonly used accumulated payoff scheme is not invariant with respect to shifts of 
payoff values when applied locally in degree-inhomogeneous population structures. We 
propose a suitably modified payoff scheme and we show both formally and by numerical 
simulation, that it leaves the replicator dynamics invariant with respect to affine transforma- 
tions of the game payoff matrix. We then show empirically that, using the modified payoff 
scheme, an interesting amount of cooperation can be reached in three paradigmatic non- 
cooperative two-person games in populations that are structured according to graphs that 
have a marked degree inhomogeneity, similar- to actual graphs found in society. The three 
games are the Prisoner's Dilemma, the Hawks-Doves and the Stag-Hunt. This confirms 
previous important observations that, under certain conditions, cooperation may emerge in 
such network-structured populations, even though standard replicator dynamics for mixing 
populations prescribes equilibria in which cooperation is totally absent in the Prisoner's 
Dilemma, and it is less widespread in the other two games. 

Key words: evolutionary games, replicator dynamics, complex networks, structured 
populations. 
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1 Introduction and Previous Work 

Evolutionary game theory (EGT) is an attempt to study the conflicting objectives 
among agents playing non-cooperative games by using Darwinian concepts related 
to frequency-dependent selection of strategies in a population fi 11213 1. instead of 
positing mathematically convenient but practically unrealistic conditions of agent 
rationality and common knowledge as is customary in classical game theory dH. 
Two concepts play a prominent role in EGT: the first is the idea of an evolution- 
arily stable strategy (ESS) and the second is the set of equations representing the 
dynamical system called replicator dynamics (RD) [ 5 1 . Both concepts are related to 
an ideal situation in which there are random independent encounters between pairs 
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of anonymous memoryless players using a given strategy in an infinite population. 
In such a situation, a strategy is said to be an ESS if a population using that strategy 
cannot be invaded by a small amount of mutant players using another strategy (this 
idea can be expressed in rigorous mathematical terms, see L2J). However, the ESS 
concept has a static character, i.e. it can be applied only once the population has 
reached a robust rest point following certain dynamics. In other words, an ESS is 
restricted to the analysis of a population in which all the members play the same 
strategy and the stability of the strategy is gauged against the invasion of a small 
amount of individuals playing another strategy. The replicator dynamics, on the 
other hand, given an initial population in which each strategy is present with some 
frequency, will end up in attractor states, as a result of the preferential selection and 
replication of certain strategies with respect to others. Simply stated, strategies that 
do better than the average will increase their share in the population, while those 
that do worse than the average will decline. The link with standard game theory is 
the following: the ESSs for a game, if at least one exists, is a subset of the game- 
theoretic equilibria called Nash equilibria (NE). The attractor states of the dynamics 
may be fixed points, cyclical attractors, or even chaotic attractors in some situation. 
However, a result of replicator dynamics guarantees that, among the rest points of 
the RD, one will find the NE and thus, a fortiori, the game's ESSs [2J. These re- 
sults pertain to infinite populations under standard replicator dynamics; they are not 
necessarily true when the assumptions are not the same e.g., finite populations with 
local interactions and discrete time evolution, which is the case considered here. 

Several problems arise in EGT when going from very large to finite, or even small 
populations which are, after all, the normal state of affairs in real situations. For ex- 
ample, in small populations theoretical ESS might not be reached, as first observed 
by Fogel et al. ffGrTI and Ficici et al. [8], and see also [9|. The method affecting the 
selection step can also be a source of difference with respect to standard EGT, even 
for infinite mixing populations. Recently, Ficici et al. (Tpl have shown that using 
selection methods different from payoff proportionate selection, such as trunca- 
tion, tournament or ranking leads to results that do not converge to the game theory 
equilibria postulated in standard replicator dynamics. Instead, they find different 
non-Nash attractors, and even cyclic and chaotic attractors. 

While the population structure assumed in EGT is panmictic, i.e. any player can 
be chosen to interact with any other player, it is clear that "natural" populations in 
the biological, ecological, and socio-economical realms often do have a structure. 
This can be the case, for instance, for territorial animals, and it is even more com- 
mon in human interactions, where a given person is more likely to interact with 
a "neighbor", in the physical or relational sense, rather than with somebody else 
that is more distant, physically or relationally. Accordingly, EGT concepts have 
been extended to such structured populations, starting with the pioneering works 
of Axelrod [llj and Nowak and May [12J who used two-dimensional grids which 
are regular lattices. However, today it is becoming clear that regular lattices are 
only approximations of the actual networks of interactions one finds in biology and 
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society. Indeed, it has become apparent that many real networks are neither reg- 
ular nor random graphs; instead, they have short diameters, like random graphs, 
but much higher clustering coefficients than the latter, i.e. agents are locally more 
densely connected. These networks are collectively called small-world networks 
(see IHBIMH ). Many technological, social, and biological networks are now known 
to be of this kind. Thus, research attention in EGT has recently shifted from mixing 
populations, random graphs, and regular lattices towards better models of social 
interaction structures fil5il6il7ll8L 

Fogel et al. ['6'7'| and Ficici et al. HIOISII studied the deviations that occur in EGT 
when some of the standard RD assumptions are not fully met. In this paper we 
would like to address another problem which arises when using RD in network- 
structured populations. In the standard setting, populations are panmictic, i.e. any 
agent may interact with any other agent in the population. However, in complex 
networks, players may have a widely different number of neighbors, depending 
on the graph structure of the network interactions. On the other hand, panmictic 
populations may be modeled as complete graphs, where each vertex (agent) has 
the same number of neighbors (degree). The same is true for any regular graph, 
and thus for lattices, and also, at least in a statistical sense, for Erdos-Renyi ran- 
dom graphs [19 1, which have a Poissonian degree distribution. In the cases where 
the number of neighbors is the same for all players, after each agent has played 
the game with all of its neighbors, one can either accumulate or average the pay- 
off earned by a player in order to apply the replicator dynamics. Either way, the 
result is the same except for a constant multiplicative factor. However, when the 
degrees of agents differ widely, these two ways of calculating an agent's payoff 
give very different results, as we show in this paper. Furthermore, we show that 
when using accumulated payoff, the RD is not invariant with respect to a positive 
affine transformation of the payoff matrix as it is prescribed by the standard RD 
theory [[2j|. In other words, the game depends on the particular payoff values and is 
non-generic f20\. Finally, we propose another way of calculating an agent's payoff 
that both takes into account the degree inhomogeneity of the network and leaves 
the RD invariant with respect to affine transformations of the payoff matrix. We 
illustrate the mathematical ideas with numerical simulations of three well-known 
games: the Prisoner's Dilemma, the Hawk-Dove, and the Stag-Hunt which are uni- 
versal metaphors for conflicting social interactions. 

In the following, we first briefly present the games used for the simulations. Next, 
we give a short account of the main population graph types used in this work, 
mainly for the sake of making the paper self-contained. Then we describe the par- 
ticular replicator dynamics that is used on networks, followed by an analysis of the 
influence of the network degree inhomogeneity on an individual's payoff calcula- 
tion. The ensuing discussion of the results of many numerical experiments should 
help illuminate the theoretical points and the proposed solutions. Finally, we give 
our conclusions. 
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2 Three Symmetric Games 



The three representative games studied here are the Prisoner's Dilemma (PD), the 
Hawk-Dove (HD), and the Stag-Hunt (SH) which is also called the Snowdrift Game 
or Chicken. For the sake of completeness, we briefly summarize the main features 
of these games here; more detailed accounts can be found in many places, for in- 
stance [.11 .2 1.22 1 . These games are all two-person, two-strategy, symmetric games 
with the payoff bi-matrix of Table [T] In this matrix, R stands for the reward the 
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Table 1 

Generic payoff bi-matrix for the two-person, symmetric games discussed in the text. 

two players receive if they both cooperate (C), P is the punishment for bilateral 
defection {D), and T is the temptation, i.e. the payoff that a player receives if it 
defects, while the other cooperates. In this case, the cooperator gets the sucker's 
payojf S. In the three games, the condition 2R > T + is imposed so that mutual 
cooperation is preferred over an equal probability of unilateral cooperation and de- 
fection. For the PD, the payoff values are ordered numerically in the following way: 
T > R > P > S. Defection is always the best rational individual choice; (D, D) 
is the unique NE and also an ESS [2J. Mutual cooperation would be preferable but 
it is a strongly dominated strategy. 

In the Hawk-Dove game, the order of P and S is reversed yielding T > R > S > 
P. Thus, in the HD when both players defect they each get the lowest payoff. (C, D) 
and {D, C) are NE of the game in pure strategies, and there is a third equilibrium in 
mixed strategies where strategy D is played with probability p, and strategy C with 
probability l—p, where p depends on the actual payoff values. The only ESS of the 
game is the mixed strategy, while the two pure NE are not ESSs [|2]|. The dilemma in 
this game is caused by "greed", i.e. players have a strong incentive to "bully" their 
opponent by playing D, which is harmful for both parties if the outcome produced 
happens to be {D,D). 

In the Stag-Hunt, the ordering isR>T>P>S, which means that mutual 
cooperation (C, C) is the best outcome, Pareto- superior, and a NE. However, there 
is a second NE equilibrium where both players defect (D, D) which is inferior 
from the Pareto domination point of view, but it is less risky since it is safer to play 
D when there is doubt about which equilibrium should be selected. From a NE 
standpoint, however, they are equivalent. Here the dilemma is represented by the 
fact that the socially preferable coordinated equilibrium (C, C) might be missed 
for "fear" that the other player will play D instead. There is a third mixed-strategy 
NE in the game, but it is commonly dismissed because of its inefficiency and also 
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because it is not an ESS []2l. 



3 Network Types 

For our purposes here, a network will be represented as an undirected graph G(y, E), 
where the set of vertices V represents the agents, while the set of edges E repre- 
sents their symmetric interactions. The population size is the cardinality of V. A 
neighbor of an agent i is any other agent j such that there is an edge {ij} E E. The 
cardinality of the set of neighbors Vi of player i is the degree ki of vertex i E V. 
The average degree of the network will be called k. An important quantity that will 
be used in the following is the degree distribution function (DDF) of a graph P{k) 
which gives the probability that a given node has exactly k neighbors. 

To expose the technical problems and their solution, we shall investigate three main 
graph population structures: regular lattices, random graphs, and scale-free graphs. 
These graph types represent the typical extreme situations studied in the literature. 
Regular lattices are examples of degree-homogeneous networks, i.e. all the nodes 
have the same number of neighbors; they have been studied from the EGT point of 
view in [I12I23I24I25II . among others. In random graphs the degree fluctuates around 
the mean k but the fluctuations are small, of the order of the standard deviation of 
the associated Poisson distribution. The situation can thus be described in mean- 
field terms and is similar to the standard setting of EGT, where the large mixing 
population can be seen as a completely connected graph. On the other hand, scale- 
free graphs are typical examples of degree-heterogeneous graphs as the degree dis- 
tribution is broad (see below). For the sake of illustration, examples of these three 
population network types are shown in Fig. [T| For random and scale-free graphs 
only one among the many possible realizations is shown, of course. 

Recent work [fT4ll has shown that scale-free and other small- world graphs are struc- 
turally and statistically much closer to actual social and biological networks and are 
thus an interesting case to study. Evolutionary games on scale-free and other small- 
world networks have been investigated, among others, in [1 511611712611 . Another 
interesting result for evolutionary games on networks has been recently obtained by 
Ohtsuki et al. [|27l . In this study the authors present a simple rule for the evolution 
of cooperation on graphs based on cost/benefit ratios and the number of neigh- 
bors of a given individual. This result is closely related to the subject matter of the 
present work but its application in the present context will be the subject of further 
study. Our main goal is to consider the global influence of network structure on the 
dynamics using a particular strategy update rule. A further step toward real social 
structures has been taken in [|T8l . where some evolutionary games are studied using 
model social networks and an actual coauthorship network. 

The DDF of a regular graph is a normalized delta function centered at the constant 
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(c) 

Fig. 1. A regular lattice (a), a random graph (b), and a scale-free graph (c). In (c) the nodes 
are shown with a size proportional to their number of neighbors. 

degree k of the graph. Random graphs, which behave similar to panmictic popu- 
lations, are constructed according to the standard Erdos-Renyi [19| model: every 
possible edge among the N vertices is present with probability p or is absent with 
probabililty 1 — p. The DDF of such a random graph is Poissonian for oo. 
Thus most vertices have degrees close to the mean value k. In contrast, DDFs for 
complex networks in general have a longer tail to the right, which means that nodes 
with many neighbors may appear with non-negligible probability. An extreme ex- 
ample are scale-free networks in which the DDF is a power-law P{k) oc k''^. 
Scale-free networks have been empirically found in many fields of technology, so- 
ciety, and science [fT4|. To build scale-free networks, we use the model proposed by 
Barabasi and Albert [|28l . In this model, networks are grown incrementally starting 
with a small clique of mg nodes. At each successive time step a new node is added 
such that its m < niQ edges link it to m nodes already present in the graph. It is 
assumed that the probability p that a new node will be connected to node i depends 
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on the current degree ki of the latter. This is called the preferential attachment rule. 
The probability p{ki) of node i to be chosen is given by p{ki) = ki/ kj, where 
the sum is over all nodes already in the graph. The model evolves into a stationary 
network with power-law probability distribution for the vertex degree P{k) ~ k~'^, 
with 7 ~ 3. 



4 Replicator Dynamics in Networks 

The local dynamics of a player i only depends on its own strategy and on the strate- 
gies of the ki players in its neighborhood Vi. Let us call rcij the payoff player i 
receives when interacting with neighbor j. Let M be the payoff matrix correspond- 
ing to the row player. Since the games used here are symmetric the corresponding 
payoff matrix of the column player is simply Af ^, the transpose of M. For example, 
from table [T] of section |2]one has: 











M = 












p) 



where suitable numerical values must be replaced for R, S, T, P. 
This payoff iiij of the row player is now defined as 

TT,j{t) = Si{t) M sj{t), 

where Si(t) and sj{t) are, respectively, row and column vectors representing the 
players' mixed strategies i.e., the probability distributions over the rows or columns 
played by i and j at time t. A pure strategy is the particular case in which only one 
row or column is chosen. The quantity 

is the accumulated payoff collected by player i at time step t, whereas the quantity 

ni(t) = ^Ili(t) is his average payoff. 

Accumulated payoff seems more logical in degree-heterogeneous networks such 
as scale-free graphs since it reflects the very fact that players may have different 
numbers of neighbors in the network. Average payoff, on the other hand, smooths 
out the possible differences although it might be justified in terms of number of 
interactions that a player may sustain in a given time. For instance, an individual 
with many connections is likely to interact less often with each of its neighbors than 
another that has a lower number of connections. Also, if there is a cost to maintain 
a relationship, average payoff will roughly capture this fact, while it will be hidden 
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if one uses accumulated payoff. On the other hand, if in a network some individu- 
als happen to have many more connections than the majority, this also means that 
they have somehow been able to establish and maintain them; maybe this is a result 
of better social skills, more opportunities or for other reasons but it is something 
that is commonly observed on actual social networks. Because of this, most re- 
cent papers dealing with evolutionary games on networks have used accumulated 
payoff [ 1511612612911811 . and this is the main reason why we have focused on the 
technical problems that this may cause in degree-heterogeneous networks. 

The rule according to which agents update their strategies is the conventional RD. 
The RD rule in networks aims at maximal consistency with the original evolution- 
ary game theory equations and is the same as proposed by [25]. It is assumed that 
the probability of switching strategy is a monotonic increasing function </) of the 
payoff difference [12J|3|. To update the strategy of player i, another player j is first 
drawn uniformly at random from i's neighborhood Vi. Then, strategy Si is replaced 
by Sj with probability 

Pi = 0(n,— n,), (1) 

Where 11 may stand either for the above defined accumulated fl or average IT pay- 
offs, or for the modified accumulated payoff IT to be defined below. The major dif- 
ference with standard replicator dynamics is that two-person encounters between 
players are only possible among neighbors, instead of being drawn from the whole 
population. Other commonly used strategy update rules include imitating the best 
in the neighborhood, or replicating in proportion to the payoff, meaning that each 
individual i reproduces with probability pi = Hi/Y^j ^Tj, where pii is i's payoff 
and the sum is over all i's neighbors [ |25l . However, in the present work we do 
not examine these alternative rules. Finally, contrary to [fT6ll , we use asynchronous 
dynamics in the simulations presented here. More precisely, we use the discrete 
update dynamics that makes the least assumption about the update sequence: the 
next node to be updated is chosen at random with uniform probability and with 
replacement. This asynchronous update is analogous to the one used by Hauert et 
al. [|25]| . It corresponds to a binomial distribution of the updating probability and is 
a good approximation of a continuous-time Poisson process. We believe that asyn- 
chronous update dynamics are more likely in a system of independently interacting 
agents that may act at different and possibly uncorrelated times. Furthermore, it has 
been shown that asynchronous updating may give rise to steadier quasi-equilibrium 
states by eliminating artificial effects caused by the nature of perfect synchronic- 
ity [l30l . Nevertheless, in this work, we have checked that synchronous update of 
the agents' strategies does not qualitatively change the conclusions. 

4. 1 Payoff Invariance 

In standard evolutionary game theory one finds that replicator dynamics is invariant 
under positive affine transformations of payoffs with merely a possible change of 
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time scale flU. Unfortunately, on degree -heterogenous networks, this assumption 
is not satisfied when combining replicator dynamics together with accumulated 
payoff. This can be seen as follows. Let pi in Eq. [l]be given by the following 
expression, as defined by Santos and Pacheco [|T6ll . 



Pi 



0(n, - n, 







if n, - n, > 



otherwise. 



(2) 



with du = max{T, R, P, S} — min{T, R, P, S}, A;> = max{ki, kj}, and Ilj 
(respectively Uj) the aggregated payoff of a player i (respectively j). If we set 
= fix for all X G y and now apply a positive affine transformation of the payoff 
matrix, this leads to the new aggregated payoff 

&i= ^'ij = ("^u + P) = o^J^'^ij + J2 = + Pki 
with a > 0, P E M. and hence 



(f>{fl'j - U'i) = {aUj + pkj - aUi - (3ki)/{adMk>) 
= (f){f[j - Ui) + (3{kj - ki)/{adMk>). 

One can clearly see that using accumulated payoff does not lead to an invariance of 
the replicator dynamics under shifts of the payoff matrix. 

As for the average payoff, although it respects the replicator dynamics invariance 
under positive affine transformation, it prevents nodes with many edges to have po- 
tentially a higher payoff than those with only a few links. Furthermore, nodes are 
extremely vulnerable to defecting neighbors with just one link. 
Thus, we propose here a third definition for a player's payoff that retains the advan- 
tages of the accumulated and average payoff definitions without their drawbacks. 
Let TT^ denote the guaranteed minimum payoff a player can obtain in a one-shot 
two-person game. This is what a player would at least receive were he to attempt 
to maximize his minimum payoff. For example in the PD, a player could choose 
to play C with the risk of obtaining the lowest payoff 5* were its opponent to play 
D. However, by opting for strategy D a player would maximize its minimum pay- 
off thus guaranteeing itself at least tt^ = P > S no matter what its opponent's 
strategy might be. In the HD game we have tt^ = S, for this time the payoff or- 
dering is T > R > S > P and a player needs only to play C to receive at least 
payoff S. Finally, in the SH game, tt^ = P. We can now define a player i's ag- 
gregated payoff as being fij = Y.j&Vi i'^ij ~ ^7)- Intuitively, it can be viewed as 
the difference between the payoff an individual collects and the minimum payoff it 
would get by "playing it safe". Our modified payoff fi has the advantage of leav- 
ing the RD invariant with respect to a positive affine transformation of the payoff 
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matrix both on degree-homogeneous and heterogeneous graphs while still allow- 
ing the degree distribution of the network to have a strong impact on the dynamics 
of the game. Indeed, a player placed on a highly connected node of a graph can 
benefit from its numerous interactions which enables it to potentially collect a high 
payoff. However, these same players run the risk of totaling a much lower score 
than a player with only a few links. One can notice that on degree-homogeneous 
graphs such as lattices or complete graphs, using accumulated, average, or the new 
aggregated payoff definition yields the same results. The proof of the RD invari- 
ance under positive affine transformation of the payoff matrix when using this new 
payoff definition is straightforward: 



0(n' - n'j 



aduk 



M'^> Vfcev, 



-a (TTjfc - TT^) 

:(n, -n,)/(rfMA;>) 
:(/.(n,-ni). 



4. 2 Modified Replicator Dynamics 



Let us turn our attention once again to the replicator dynamics rule (Eqj2]). Dividing 
the payoff difference between players j and i by c/a/A;> might seem reasonable at 
first since it does ensure that is a probability, i.e. has a value between and 1. 
Nevertheless, we don't find it to be the adequate division to do for subtle reasons. 
To illustrate our point, let us focus on the following particular case and use the 
accumulated payoff to simplify the explanation. 

On the one side. Fig. [2] (a) shows a cooperator CI surrounded by three defectors 
each having three cooperating neighbors. Using the replicator dynamics as defined 
in Eq. [2| the probability cooperator CI would turn into a defector, given that it is 
selected to be updated, is equal to 
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(a) 



(b) 



Fig. 2. Example 

(/.(n, - fici) = (n, - fid)/ {dKiK) 

= i3T-3S)/i3dM) 
= (T - S)/dM, 

and this no matter which defecting neighbor j is chosen since they all have the 
same payoff. On the other side, the central cooperator C2 in Fig.[2](b) would adopt 
strategy D with probability 



0(n, - flea) = (fij - flc2)/{dMk>) 
= (3T - 6S)/QdM 
= {T-2S)/2dM, 

a value that is once again independent of the selected neighbor j. Now, if T > 
and 0(nj — fici), 0(5^ — nc2) > 0, then C2 has a bigger chance of having its 
strategy unaltered than CI does. This last statement seems awkward since in our 
opinion, the fact of being surrounded by twice as many defectors as CI (with all 
the D-neighbors being equally strong), should have a negative impact on coopera- 
tor C2, making it difficult for it to maintain its strategy. To make the situation even 
more evident, let us also suppose S* = 0. In this case, a cooperator surrounded by 
an infinite number of D-neighbors, who in turn all have a finite number of neigh- 
bors, would have a zero probability of changing strategy, which is counter-intuitive. 
Therefore, and with all the previous arguments in mind, we adjust Eq. [2]to define 
another replicator dynamics function namely 



0(n, - Ui 



nj,max ni 







if Uj - Hi > 



otherwise. 



(3) 



where Ilx,msix (resp. H^; min) is the maximum (resp. minimum) payoff a player x can 
get. If vr^ max and vr^, ^in denote player x's maximum and minimum payoffs in a two- 
player one-shot game (nx^max = max{T, R, P, 5"} and vTa- = min{T, R, P, 5*} 
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for the dilemmas studied here), we have: 



• ^x,max = vra;,max and Il^^min = T^x,mm for average payoff; 

• ^x,max = fcxvrx,max and U^^^min = KT^x,mm for accumulated payoff; 

• n^^n^ax = K{T^x,m?.x " T^x^) ^nd = K{Tfx,mi^ " '^x,-^) for the new payoff 
scheme. 

Finally, one can easily verify that using Ilj = Hj as the aggregated payoff of a 
player i leaves equation Eq.[3]invariant with respect to a positive affine transforma- 
tion of the payoff matrix. 




Fig. 3. Amount of cooperation in the HD game using accumulated payoff on three different 
network types in three different game spaces (see text). Lighter areas mean more coopera- 
tion than darker ones (see scale on the right side). Left column: scale free; Middle column: 
random graph; Right column: grid. Upper row: 2<r<3, i? = 2, 1<5'<2, P = l; 
Middle row: 1<T<2, i? = l, 0<5<1, P = 0; Bottom row: < T < 1, ii = 0, 
-1<5<0, P = -l 
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5 Numerical Simulations 




(a) (b) 

Fig. 4. Standard deviation for the HD using accumulated payoff on scale-free networks for 
two different game spaces, (a) 1 < T < 2, = 1, 5 = 0.1, P = 0, (b) 2 < T < 3, i? = 2, 
S = 1.1, P = 1. Note that (a) is a cut at S" = 0.1 of the middle image in the leftmost 
column of Fig. |3] while (b) represents a cut of the topmost image in the leftmost column of 
Fig.[3]atS' = 1.1. 

We have simulated the PD, HD and SH described in Sect. [2] on regular lattices, 
Erdos-Renyi random graphs and Barabasi-Albert scale-free graphs, aU three of 
which were presented in Sect. [3] Furthermore, in each case, we test the three payoff 
schemes discussed in Sect. |4l The networks used are all of size = 4900 with 



shift -1 scale free no shift scale free shift +1 scale free 




s s s 



Fig. 5. Levels of cooperation in the HD game using the new aggregated payoff 11 on 
scale-free graphs in three different game spaces (see text). Left: 2 < T < 3, i? = 2, 
1 < 5 < 2, P = 1; Middle: l<r<2, P=l, 0<S'<1, P = 0; Right: < T < 1, 
P = 0, -1 < S" < 0, P = -1. 

an average degree k = A. The regular lattices are two-dimensional with periodic 
boundary conditions, and the neighborhood of an individual comprises the four 
closest individuals in the north, east, south, and west directions. The Erdos-Renyi 
random graphs were generated using connection probability p = 8.16 x 10^'^. Fi- 
nally, the Barabasi-Albert were constructed starting with a clique of niQ = 2 nodes 
and at each time step the new incoming node has m = 2 links. 
For each game, we limit our study to the variation of only two parameters per 
game. In the case of the PD, we set i? = 1 and S' = 0, and vary 1 < T < 2 and 
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< P < 1. For the HD, we set i? = 1 and P = and the two parameters are 

1 < T < 2 and < 5 < 1. Finally, in the SH, we decide to fix i? = 1 and S = 
and vary < T < 1 and < P < T. 

We deliberately choose not to vary the same two parameters in all three games. The 
reason we choose to set T and S in both the PD and the SH is to simply provide 
natural bounds on the values to explore of the remaining two parameters. In the 
PD case, P is limited between P = 1 and = in order to respect the ordering 
of the payoffs {T > R > P > S) and T's upper bound is equal to 2 due to the 
2P > T + S constraint. In the HD, setting P = 1 and P = determines the range 
of S (since this time T > R > S > P) and gives an upper bound of 2 for T, again 
due to the 2P > T + S constraint. Note however, that the only valid value pairs of 
(T, S) are those that satisfy the latter constraint. Finally, in the SH, both T and P 
range from S to P. Note that in this case, the only valid value pairs of (T, P) are 
those that satisfy T > P. 

It is important to realize that, when using our new aggregated payoff or the average 
payoff, even though we reduce our study to the variation of only two parameters 
per game, we are actually exploring the entire game space. This is true owing to the 
invariance of Nash equilibria and replicator dynamics under positive affine transfor- 
mations of the payoff matrix [2J. As we have shown earlier and as we will confirm 
numerically in the next section, this does not hold for the accumulated payoff. 
Each network is randomly initialized with exactly 50% cooperators and 50% de- 
fectors. In all cases, the parameters are varied between their two bounds by steps of 
0.1. For each set of values, we carry out 50 runs of 15000 time steps each, using a 
fresh graph realization in each run. Cooperation level is averaged over the last 1000 
time steps, well after the transient equilibration period. In the figures that follow, 
each point is the result of averaging over 50 runs. In the next two sections, in order 
to avoid overloading this document with figures, we shall focus each time on one 
of the three games, commenting on the other two along the way. 



5.1 Payoff Shift 

We have demonstrated that in theory, the use of accumulated payoff does not leave 
the RD invariant under positive affine transformations of the payoff matrix. How- 
ever, one can wonder whether in practice, such shifts of the payoff matrix translate 
into significant differences in cooperation levels or are the changes just minor. 

Figure [3] depicts the implications of a slight positive and negative shift of the HD 
payoff matrix. As one can clearly see, the cooperation levels encountered are no- 
tably different before and after the shift. As a matter of fact, when comparing be- 
tween network types, scale-free graphs seem to do less well in terms of cooperation 
than regular grids with a shift of —1, and not really better than random graphs with 
a shift of +1. Thus, one must be extremely cautious when focusing on a rescaled 
form of the payoff matrix, affirming that such a re-scaling can be done without loss 
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Fig. 6. Levels of cooperation in the PD game space using three different payoff schemes 
and two different network types. Left column: Accumulated Payoff; Middle column: New 
Aggregated Payoff; Right column: Average Payoff. Upper row: Scale free graph; Bottom 
row: Random graph. Game space: 1<T<2, i? = l,0<P< 1,5 = 0. 



of generality, for this is far from true when dealing with accumulated payoff. 
The noisy aspect of the top two figures of the leftmost column of Fig. |3]has caught 
our attention. It is essentially due to the very high standard deviation values we find 
in the given settings (see Fig.|4]). This observation is even more pronounced with a 
shift of +1. This shows that replicator dynamics becomes relatively unstable when 
using straight accumulated payoff. 



We have run simulations using our payoff 11, on all three network types in order to 
numerically validate the invariance of the RD with this payoff scheme. However, 
to save space, we only show here the results obtained on scale-free graphs which 
are the networks that generated the biggest differences in the accumulated payoff 
case (see Fig.js} leftmost colummn). As one can see in Fig.js} using ft does indeed 
leave the RD invariant with respect to a shift of the payoff matrix. There are minor 
differences between the figures, but these are simply due to statistical sampling and 
roundoff errors. Finally, a shift of the payoff matrix has, as expected, no influence 
at all on the general outcome when using the average payoff. We point out that the 
same observations can also be made for the PD and SH cases (not shown here). 
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5.2 Payoff and Network Influence on Cooperation 



In this section we report results on global average cooperation levels using the three 
payoff schemes for two games on scale-free and random graphs. 
Figure [6]illustrates the cooperation levels reached for the PD game, in the 1 < T < 
2, i?=l, 0<P<1, S' = game space, on a Barabasi-Albert scale-free and ran- 
dom graphs, and when using each of the three different payoff schemes mentioned 
earlier, namely 11, 11 and 11. 

We immediately notice that there is a significant parameter zone for which accu- 
mulated payoff (leftmost column) seems to drastically promote cooperation com- 
pared to average payoff (rightmost column). This observation has already been 
highlighted in some previous work [I30I29II , although it was done for a reduced 
game space. We nevertheless include it here to situate the results obtained using 
our adjusted payoff in this particular game space in comparison to those obtained 
using the two other extreme payoff schemes. On both network types, 11 (central 
column of Fig. |6]) yields cooperation levels somewhat like those obtained with ac- 
cumulated payoff but to a lesser degree. This is especially striking on scale-free 
graphs (upper row of Fig. [6]). However, we again point out that the situation shown 
in the image of the upper left comer of Fig. |6] would change dramatically under a 
payoff shift, as discussed in Sect. |5.1| for the HD game. The same can be observed 
for the HD and SH games (see Fig. |7]for the SH case). On regular lattices, there are 
as expected no differences whatsoever between the use of 11 over 11 or 11 due to the 
degree homogeneity of this type of network (not shown). 

The primary goals of this work are to highlight the non-invariance of the RD un- 
der affine transformations of the payoff matrix when using accumulated payoff, 
and to propose an alternative payoff scheme without this drawback. How does the 
network structure influence overall cooperation levels when this latter payoff is 
chosen? Looking at the middle column of figures [6] and |7| we observe that de- 
gree non-homogeneity enhances cooperation. The relatively clear separation in the 
game space between strongly cooperative regimes and entirely defective ones in 
the middle column of Fig. |7| which refers to the SH game, can be explained by 
the existence of the two ESSs in pure strategies in this case. Similarly, the large 
transition phase from full cooperation to full defect states in the HD (middle image 
of Fig. [5]) is due to the fact that the only ESS for this game is a mixed strategy. 
Cooperation may establish and remain stable in networks thanks to the formation of 
clusters of cooperators, which are tightly bound groups of players. In the scale-free 
case this is easier for, as soon as a highly connected node becomes a cooperator, 
if a certain number of its neighbors are cooperators as well, chances are that all 
neighbors will imitate the central cooperator, which is earning a high payoff thanks 
to the number of acquaintances it has. An example of such a cluster is shown in 
Fig.[8]for the PD. A similar phenomenon has been found to underlie cooperation in 
real social networks [.18J . 
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Fig. 7. Cooperation levels for the SH game space using three different payoff schemes 
and two different network types. Left column: Accumulated Payoff; Middle column: New 
Aggregated Payoff; Right column: Average Payoff. Upper row: Scale free graph; Bottom 
row: Random graph. Game space: i? = l, 0<T<1, 0<P<1, S' = 0. Note that the 
meaningful game space is the upper left triangle, i.e. when T > P. 

In order to explore the dependence of the evolutionary processes on the network 
size, we have performed simulations with two other graph sizes (N = 2450 and 
= 9800) for the HD game. To save space, we do not show the figures but coop- 
eration results are qualitatively very similar to those shown here for = 4900. We 
have also simulated populations with two different initial percentages of randomly 
distributed cooperators: 30% and 70%; again, there are no qualitative differences 
with the 50-50 case shown here. 



6 Conclusions 

Standard RD assumes infinite mixing populations of playing agents. Actual and 
simulated populations are necessarily of finite size and show a network of ties 
among agents that is not random, as postulated by the theory. In this work we 
have taken the population finiteness for granted and we have focused on the graph 
inhomogeneity aspects of the problem. It is a well known fact that agent cluster- 
ing may provide the conditions for increased cooperation levels in games such as 
those studied here. However, up to now, only regular structures such as grids had 
been studied in detail, with the exception of a few investigations that have dealt with 
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Fig. 8. A cluster with a majority of cooperators (triangles) with many links to a central 
cooperator. Symbol size is proportional to degree. Links to other nodes of the network have 
been suppressed for clarity. 

small- world population structures of various kinds [I15I16I17I27I18L But most have 
used an accumulated payoff scheme that makes no difference in regular graphs, but 
in the other cases, it does not leave the RD invariant with respect to affine trans- 
formations of the payoff matrix, which is required by evolutionary game theory. 
This gives rise to results that are not generalizable to the whole game space. The 
alternative of using average payoff respects invariance but is much less realistic 
in degree-inhomogeneous networks that are the rule in society. Here we have pro- 
posed a new payoff scheme that correctly accounts for the degree inhomogeneity of 
the underlying population graph and, at the same time, is invariant with respect to 
these linear transformations. Using this scheme, we have shown that, on complex 
networks, cooperation may reach levels far above what would be predicted by the 
standard theory for extended regions of the game's parameter space. The emergence 
of cooperation is essentially due to the progressive colonization by cooperators of 
highly connected clusters in which linked cooperators that earn a high payoff mu- 
tually protect themselves from exploiting defectors. This phenomenon had already 
been observed to a lesser extent in populations structured as regular grids but it 
is obviously stronger for scale-free graphs, where there exist a sizable number of 
highly connected individuals and it is the same effect that underlies cooperation in 
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actual social networks. This observation alone may account for observed increased 
levels of cooperation in society without having to take into account other factors 
such as reputation, belonging to a recognizable group, or repeated interactions giv- 
ing rise to complex reciprocating strategies, although these factors also play a role 
in the emergence of cooperation. 
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